Discriminating Acute Respiratory Distress Syndrome from other forms of respiratory failure via iterative machine learning.

Acute Respiratory Distress Syndrome (ARDS) is associated with high morbidity and mortality. Identification of ARDS enables lung protective strategies, quality improvement interventions, and clinical trial enrolment, but remains challenging particularly in the first 24 hours of mechanical ventilation. To address this we built an algorithm capable of discriminating ARDS from other similarly presenting disorders immediately following mechanical ventilation. Specifically, a clinical team examined medical records from 1263 ICU-admitted, mechanically ventilated patients, retrospectively assigning each patient a diagnosis of "ARDS" or "non-ARDS" (e.g., pulmonary edema). Exploiting data readily available in the clinical setting, including patient demographics, laboratory test results from before the initiation of mechanical ventilation, and features extracted by natural language processing of radiology reports, we applied an iterative pre-processing and machine learning framework. The resulting model successfully discriminated ARDS from non-ARDS causes of respiratory failure (AUC = 0.85) among patients meeting Berlin criteria for severe hypoxia. This analysis also highlighted novel patient variables that were informative for identifying ARDS in ICU settings.

Higher accuracy of complex medication reconciliation through improved design of electronic tools.

Objective: Investigate the accuracy of 2 different medication reconciliation tools integrated into electronic health record systems (EHRs) using a cognitively demanding scenario and complex medication history. Materials and Methods: Seventeen physicians reconciled medication lists for a polypharmacy patient using 2 EHRs in a simulation study. The lists contained 3 types of discrepancy and were transmitted between the systems via a Continuity of Care Document. Participants updated each EHR and their interactions were recorded and analyzed for the number and type of errors. Results: Participants made 748 drug comparisons that resulted in 53 errors (93% accuracy): 12 using EHR2 (3% rate, 0-3 range) and 41 using EHR1 (11% rate, 0-9 range; P < .0001). Twelve clinicians made completely accurate reconciliations with EHR2 (71%) and 6 with EHR1 (35%). Most errors (28, 53%) occurred in medication entries containing discrepancies: 4 in EHR2 and 24 in EHR1 (P = .008). The order in which participants used the EHRs to complete the task did not affect the results. Discussion: Significantly fewer errors were made with EHR2, which presented lists in a side-by-side view, automatically grouped medications by therapeutic class and more effectively identified duplicates. Participants favored this design and indicated that they routinely used several workarounds in EHR1. Conclusion: Accurate assessment of the safety and effectiveness of electronic reconciliation tools requires rigorous testing and should prioritize complex rather than simpler tasks that are currently used for EHR certification and product demonstration. Higher accuracy of reconciliation is likely when tools are designed to better support cognitively demanding tasks.

Clinical reasoning in the context of active decision support during medication prescribing.

OBJECTIVE: Describe and analyze reasoning patterns of clinicians responding to drug-drug interaction alerts in order to understand the role of patient-specific information in the decision-making process about the risks and benefits of medication therapy. Insights could be used to inform the design of decision-support interventions. METHODS: Thirty-two clinicians working with five EHRs in two countries completed sets of six medication orders each and responded to high- and low-severity drug-drug interaction alerts while verbalizing their thoughts in a standard think-aloud protocol. Tasks were recorded and analyzed to describe reasoning patterns about patient-risk assessment and strategies to avoid or mitigate it. RESULTS: We observed a total of 171 prescribing decisions. Clinicians actively sought to reduce risk when responding to high-severity alerts, mostly by monitoring patients and making dose adjustments (52 alerts, 40%). In contrast, they routinely left prescriptions unchanged after low-severity alerts when they felt confident that patients would tolerate the drug combination and that treatment benefits outweighed the risks (30 alerts, 71%). Clinicians used similar reasoning patterns regardless of the EHR used and differences in alert design. DISCUSSION: Clinicians conceptualized risk as a complex set of interdependent tradeoffs specific to individual patients and had a tendency not to follow advice they considered of low clinical value. Omission of patient-specific data, which was not shown in alerts or included in trigger logic, may have contributed to the constancy of reasoning and to similarities in risk-control strategies we observed despite significant differences in interface design and system function. CONCLUSION: Declining an alert suggestion was preceded by sometimes brief but often complex reasoning, prioritizing different aspects of care quality and safety, especially when the perceived risk was higher. Clinicians believed that the risk indicated in drug-drug interaction alerts needs to be interpreted as one factor in the broader context of care, specific to a patient.

Accuracy and Completeness of Clinical Coding Using ICD-10 for Ambulatory Visits.

This study describes a simulation of diagnostic coding using an EHR. Twenty-three ambulatory clinicians were asked to enter appropriate codes for six standardized scenarios with two different EHRs. Their interactions with the query interface were analyzed for patterns and variations in search strategies and the resulting sets of entered codes for accuracy and completeness. Just over a half of entered codes were appropriate for a given scenario and about a quarter were omitted. Crohn's disease and diabetes scenarios had the highest rate of inappropriate coding and code variation. The omission rate was higher for secondary than for primary visit diagnoses. Codes for immunization, dialysis dependence and nicotine dependence were the most often omitted. We also found a high rate of variation in the search terms used to query the EHR for the same diagnoses. Changes to the training of clinicians and improved design of EHR query modules may lower the rate of inappropriate and omitted codes.

Development of a cognitive framework of patient record summary review in the formative phase of user-centered design.

Excellent usability characteristics allow electronic health record (EHR) systems to more effectively support clinicians providing care and contribute to better quality and safety. The Office of the National Coordinator for Health IT (ONC) therefore requires all vendors to follow a User-Centered Design (UCD) process to increase the usability of their products in order to meet certification criteria for the Safety-Enhanced Design part of the Meaningful Use (stage 2) EHR incentive program. This report describes the initial stage of a UCD process in which foundational design concepts were formulated. We designed a functional prototype of an EHR module intended to help clinicians to efficiently complete a summary review of an electronic patient record before an ambulatory visit. Cognitively-based studies were performed and the results used to develop a cognitive framework that subsequently guided design of a prototype. Results showed that clinicians categorized and reasoned with patient data in distinct patterns; they preferred to review relevant history in the assessment and plan section of the most recent note, to search for changes in health and for new episodes of care since the last visit and to look up current-day data such as vital signs. These basic concepts were represented in the design, for instance, by screen division into vertical thirds that had historical content to the left and most recent data to the right. Other characteristics such as visual association of contextual information or direct, one-click access to the assessment and plan section of visit notes were directly informed by our findings and refined in a series of UCD-specific iterative testing. Understanding of tasks and cognitive demands early in the UCD process was critically important for developing a tool optimized for reasoning and workflow preferences of clinicians.

Cognitive Errors in Reconciling Complex Medication Lists.

Discrepancies between multiple electronic versions of patient medication records contribute to adverse drug events. Regular reconciliation increases their accuracy but is often inadequately supported by EHRs. We evaluated two systems with conceptually different interface designs for their effectiveness in resolving discrepancies. Eleven clinicians reconciled a complex list of 16 medications using both EHRs in the same standardized scenario. Errors such as omissions to add or discontinue a drug or to update a dose were analyzed. Clinicians made three times as many errors working with an EHR with lists arranged in a single column than when using a system with side-by-side lists. Excessive cognitive effort and reliance on memory was likely a strong contributing factor for lower accuracy of reconciliation. As errors increase with task difficulty, evaluations of reconciliation tools need to focus on complex prescribing scenarios to accurately assess effectiveness, error rate and whether they reduce risk to patient safety.

Uncertainty, Case Complexity and the Content of Verbal Handoffs at the Emergency Department.

Handoffs are known to increase the risk of medical error and adverse events. Few electronic tools can support this process effectively, however. Our objective was to describe the relationship between clinical complexity, diagnostic uncertainty, fit with illness script and the content of case presentations by physicians. We observed the handoff of care for150 patients during eleven shift changes at a large urban emergency department (ED). Results indicate that as uncertainty about diagnosis and perceived illness script increased, more descriptive detail was conveyed to the incoming physicians. Physicians were concerned primarily with creating a shared mental model of a patient's clinical state and with describing the expected path to disposition rather than simply passing on data and findings. Electronic tools for ED handoffs should allow adjustment of structure and content to capture complexity and uncertainty appropriately without requiring extra effort for more routine cases that better fit to more standard narratives.

Coordination of care for complex pediatric patients: perspectives from providers and parents.

Coordinators help patients requiring complex chronic care manage frequent ambulatory visits and services received at home or from community-based agencies. EHRs directly support only a few of the required tasks as they do not allow access to all parties involved in care. Our goal was to examine how technology was used to coordinate efforts and to describe common barriers and facilitators. Insights may inform the design of tools that would effectively support identified goals. We conducted five hours of interviews with sixteen parents and six clinicians and characterized emergent themes from transcripts. Situational awareness, care and visit planning, document aggregation, abstraction and interpretation were tasks essential to coordination yet generally poorly supported by EHRs. Providers communicated primarily by email, telephone and by exchanging paper and scanned documents. A preliminary model of coordination that could be used in the planning and testing stages of a User Centered Design process is described.

Using a medical simulation center as an electronic health record usability laboratory.

Usability testing is increasingly being recognized as a way to increase the usability and safety of health information technology (HIT). Medical simulation centers can serve as testing environments for HIT usability studies. We integrated the quality assurance version of our emergency department (ED) electronic health record (EHR) into our medical simulation center and piloted a clinical care scenario in which emergency medicine resident physicians evaluated a simulated ED patient and documented electronically using the ED EHR. Meticulous planning and close collaboration with expert simulation staff was important for designing test scenarios, pilot testing, and running the sessions. Similarly, working with information systems teams was important for integration of the EHR. Electronic tools are needed to facilitate entry of fictitious clinical results while the simulation scenario is unfolding. EHRs can be successfully integrated into existing simulation centers, which may provide realistic environments for usability testing, training, and evaluation of human-computer interactions.

Design of decision support interventions for medication prescribing.

OBJECTIVE: Describe optimal design attributes of clinical decision support (CDS) interventions for medication prescribing, emphasizing perceptual, cognitive and functional characteristics that improve human-computer interaction (HCI) and patient safety. METHODS: Findings from published reports on success, failures and lessons learned during implementation of CDS systems were reviewed and interpreted with regard to HCI and software usability principles. We then formulated design recommendations for CDS alerts that would reduce unnecessary workflow interruptions and allow clinicians to make informed decisions quickly, accurately and without extraneous cognitive and interactive effort. RESULTS: Excessive alerting that tends to distract clinicians rather than provide effective CDS can be reduced by designing only high severity alerts as interruptive dialog boxes and less severe warnings without explicit response requirement, by curating system knowledge bases to suppress warnings with low clinical utility and by integrating contextual patient data into the decision logic. Recommended design principles include parsimonious and consistent use of color and language, minimalist approach to the layout of information and controls, the use of font attributes to convey hierarchy and visual prominence of important data over supporting information, the inclusion of relevant patient data in the context of the alert and allowing clinicians to respond with one or two clicks. CONCLUSION: Although HCI and usability principles are well established and robust, CDS and EHR system interfaces rarely conform to the best known design conventions and are seldom conceived and designed well enough to be truly versatile and dependable tools. These relatively novel interventions still require careful monitoring, research and analysis of its track record to mature. Clarity and specificity of alert content and optimal perceptual and cognitive attributes, for example, are essential for providing effective decision support to clinicians.

Interface design principles for usable decision support: a targeted review of best practices for clinical prescribing interventions.

Developing effective clinical decision support (CDS) systems for the highly complex and dynamic domain of clinical medicine is a serious challenge for designers. Poor usability is one of the core barriers to adoption and a deterrent to its routine use. We reviewed reports describing system implementation efforts and collected best available design conventions, procedures, practices and lessons learned in order to provide developers a short compendium of design goals and recommended principles. This targeted review is focused on CDS related to medication prescribing. Published reports suggest that important principles include consistency of design concepts across networked systems, use of appropriate visual representation of clinical data, use of controlled terminology, presenting advice at the time and place of decision making and matching the most appropriate CDS interventions to clinical goals. Specificity and contextual relevance can be increased by periodic review of trigger rules, analysis of performance logs and maintenance of accurate allergy, problem and medication lists in health records in order to help avoid excessive alerting. Developers need to adopt design practices that include user-centered, iterative design and common standards based on human-computer interaction (HCI) research methods rooted in ethnography and cognitive science. Suggestions outlined in this report may help clarify the goals of optimal CDS design but larger national initiatives are needed for systematic application of human factors in health information technology (HIT) development. Appropriate design strategies are essential for developing meaningful decision support systems that meet the grand challenges of high-quality healthcare.

Complementary methods of system usability evaluation: surveys and observations during software design and development cycles.

Poor usability of clinical information systems delays their adoption by clinicians and limits potential improvements to the efficiency and safety of care. Recurring usability evaluations are therefore, integral to the system design process. We compared four methods employed during the development of outpatient clinical documentation software: clinician email response, online survey, observations and interviews. Results suggest that no single method identifies all or most problems. Rather, each approach is optimal for evaluations at a different stage of design and characterizes different usability aspect. Email responses elicited from clinicians and surveys report mostly technical, biomedical, terminology and control problems and are most effective when a working prototype has been completed. Observations of clinical work and interviews inform conceptual and workflow-related problems and are best performed early in the cycle. Appropriate use of these methods consistently during development may significantly improve system usability and contribute to higher adoption rates among clinicians and to improved quality of care.

Analysis of user behavior in accessing electronic medical record systems in emergency departments.

Electronic patient tracking and records systems in emergency departments often connect to hospital information systems, ambulatory patient records and ancillary systems. The networked systems may not be fully interoperable and clinicians need to access data through different interfaces. This study was conducted to describe the interactive behavior of clinicians working with partially interoperable clinical information systems. We performed 78 hours of observation at two emergency departments, shadowing five physicians, ten nurses and four administrative staff. Actions related to viewing or recording data in any system or on paper were recorded. Collected data were compared along clinical roles and contrasted with findings across the two hospital sites. The findings suggest that differences in the levels of interoperability may affect the ways physicians and nurses interact with the systems. When tradeoffs in functionality are necessary for connecting ancillary systems, the effects on clinicians and staff need to be considered.

Usability testing of a Falls Prevention Tool Kit for an inpatient acute care setting.

Efforts to prevent falls in the hospital setting involves identifying patients at risk of falling and implementing fall prevention strategies. This poster describes the method and results of Performance Usability Testing on a web-based Fall Prevention Tool Kit (FPTK) developed as part of a research study, (Falls TIPS-Tailoring Interventions for Patient Safety) funded by The Robert Wood Johnson Foundation.

Technology for emergency care: cognitive and workflow considerations.

Hospitals transitioning from paper to electronic information systems often find inadequate fit between newly implemented technology and work environment causing delays, inefficient use of resources and new kind of errors attributable to problems in human interaction with computer systems. The purpose of this study is to characterize the workflow, decision making and cognitive processing of clinicians in the process of care in emergency department of a large urban hospital and to suggest possible technological interventions for identified problem areas. Through the analysis of collected data we identified fifteen tasks and areas that either slowed work progress by unnecessary duplication or created potential for error generation. Recommendations are made for the replacement of currently inadequate or non-existent technology support of clinicians by information and communication technology specifically selected to fit the cognitive and workflow demands of the task.

Comprehensive analysis of a medication dosing error related to CPOE.

This case study of a serious medication error demonstrates the necessity of a comprehensive methodology for the analysis of failures in interaction between humans and information systems. The authors used a novel approach to analyze a dosing error related to computer-based ordering of potassium chloride (KCl). The method included a chronological reconstruction of events and their interdependencies from provider order entry usage logs, semistructured interviews with involved clinicians, and interface usability inspection of the ordering system. Information collected from all sources was compared and evaluated to understand how the error evolved and propagated through the system. In this case, the error was the product of faults in interaction among human and system agents that methods limited in scope to their distinct analytical domains would not identify. The authors characterized errors in several converging aspects of the drug ordering process: confusing on-screen laboratory results review, system usability difficulties, user training problems, and suboptimal clinical system safeguards that all contributed to a serious dosing error. The results of the authors' analysis were used to formulate specific recommendations for interface layout and functionality modifications, suggest new user alerts, propose changes to user training, and address error-prone steps of the KCl ordering process to reduce the risk of future medication dosing errors.

When you come to a fork in the road, take it: strategy selection in order entry.

This study investigates how CPOE system users choose data input strategies for entering clinical orders. Complex systems often allow more than one way to complete a task. However, the appropriate entry strategy in the context of a specific clinical workflow situation may not be apparent to users. We have conducted a cognitive analysis of user interaction strategies for entering IV injection orders using a commercial CPOE system. We characterized the set of available information resources in the system interface and in the users' memory, and evaluated how effectively the application supported decision-making processes. Seven internal medicine residents participated in an experiment entering IV heparin orders to manage anticoagulation therapy. The analysis showed that efficiency was contingent upon high level of procedural and conceptual system knowledge. CPOE interface design needs to conform to decision-making and workflow processes if the technology is to become an effective clinical tool.

Computer-based drug ordering: evaluation of interaction with a decision-support system.

Provider order entry systems (POE) often incorporate active decision-support component for drug dosing. The efficacy of automated alerts that suggest dose amounts to the clinician in real time depends in part on how well they are timed to fit into the decision process and on their representational structure. We have conducted a cognitive evaluation of an interaction with a POE system that offered active decision support for heparin dosing with the goal of characterizing its effectiveness and opportunities for error. Two researchers completed a cognitive walk-through of an ordering task based on a clinical scenario. In addition, seven clinicians were asked to enter a set of orders in an experiment using the same scenario. The analysis revealed that users without a solid conceptual knowledge of the ordering system followed patterns of inefficient interactive behavior resulting in delays and some errors. Physicians often did not take full advantage of automatic dose computation provided by a decision support component and used it largely as reference. The calculated dose was not perceptually salient in the generated alert and required users to engage in meaning interpretation of the displayed information. Better visual presentation of the alert message would likely result in faster and less cognitively demanding interaction.

A framework for analyzing the cognitive complexity of computer-assisted clinical ordering.

Computer-assisted provider order entry is a technology that is designed to expedite medical ordering and to reduce the frequency of preventable errors. This paper presents a multifaceted cognitive methodology for the characterization of cognitive demands of a medical information system. Our investigation was informed by the distributed resources (DR) model, a novel approach designed to describe the dimensions of user interfaces that introduce unnecessary cognitive complexity. This method evaluates the relative distribution of external (system) and internal (user) representations embodied in system interaction. We conducted an expert walkthrough evaluation of a commercial order entry system, followed by a simulated clinical ordering task performed by seven clinicians. The DR model was employed to explain variation in user performance and to characterize the relationship of resource distribution and ordering errors. The analysis revealed that the configuration of resources in this ordering application placed unnecessarily heavy cognitive demands on the user, especially on those who lacked a robust conceptual model of the system. The resources model also provided some insight into clinicians' interactive strategies and patterns of associated errors. Implications for user training and interface design based on the principles of human-computer interaction in the medical domain are discussed.